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TEMPLATE MATCHING METHOD, TEMPLATE MATCHING 
APPARATUS, AND RECORDING MEDIUM THAT RECORDS PROGRAM 

FOR IT 

5 TECHNICAL FIELD 
[0001] 

The present invention relates to a template matching method of and 
a template matching apparatus for detecting the position of an image region 
similsu* to a template image region: from on a reference image. 

10 

BACKGROUND ART 
[0002] 

Template matching is a processing for detecting an image region 
having highest correlation to a template image region from on a reference 
15 image. The template matching is used in processings for searching a 

similar image, tracking an object, detecting motion vector in motion picture 

coding, and the like. 

[0003] 

Non-patent document 1 describes conventional template matching 
20 technologies in detail in a chapter 6. 
[0004] 

1. A method of defining a degree of similarity showing the difference 
between a partial region (reference image region) on a reference image 
nominated as a matching candidate axid a template image and finding a 
25 reference image region for minimizing the degree of similarity is ordinarily 
used as a template matching method. Used as the degree of similarity is, 
for example, the average vedue of the signal value differences of the 
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respective pixels between a reference image region and a template image 
(MAD: Mean Absolute Difference), the average value of the squares of signal 
value differences (MSE: Mean Square Error), and the like, 
[0005] 

5 As a basic method of the template matching, a method of calculating 

degrees of similarity to all the candidates and detecting a reference image 
region for minimizing them is called an all search method. Although the 
all search method can detect the reference image region that minimizes the 
degree of similarity without error, it is very ineffective from a view point of 
10 an amount of calculation. 
[0006] 

A method called a multistep search method or a rough/fine search 
method is widely used as a conventional technology for reducing the amount 
of calculation of the template matching. This is a method of dividing a 

15 template matching processing to plural steps. According to the method, at 
a first step, image regions are widely and roughly searched by evaluating 
the positions of reference image regions nominated as similar region 
candidates after the positions of them are thinned out at rough intervals, 
and, in the matching at second and subsequent steps, the similar region 

20 candidates are narrowed down stepwise by narrowly and finely searching 
only the peripheries of the image regions selected in the previous step. 
[0007] 

Fig. 2 is a schematic view showing an example of the multistep 
search method. A lattice pattern in the figure shows the coordinates of a 
25 reference image region nominated as a matching candidate. In the 

example, a template matching processing is executed at two steps. At a 
first step, the degrees of similarity of the reference image regions labeled by 
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1 in Fig. 2 are calculated, and the image region that minimizes an evaluated 
value is selected as a similar image region. In Fig. 2, when it is assumed 
that the reference image region surrounded by a circle and shown by 1 is 
selected, at a next second step, the degrees of similarity of the reference 
5 image regions labeled by 2 in Fig. 2 are calculated, and the reference image 
region that minimizes a degree of similarity in the reference image regions 
surrounded by a circle and labeled by 1 and 2 is selected as a similar image 
region. 
[0008] 

10 In the example, since 15 x 15 = 225 pieces of reference image regions 

are listed as matching candidates, when a matching processing is executed 
by the all search method, it requires to calciilate the degree of similarity 225 
times. The number of times of calculation can be reduced to 7 x 7 = 49 
times at first step and to 8 times at a second step, that is, to 57 times in 

15 total by using the multistep search method . 
[0009] 

Note that there is practically used a method of using a reference 
image reduced to low resolution at a previous step and a template image to 
further reduce the amount of calculation of the multistep search method. 

20 For example, at a first step of the example of Fig. 1, when the degree of 

similarity is calculated using an image having resolution reduced to 1/2 both 
horizontally and vertically, since the amount of calculation for calculating 
the degree of similarity once is reduced to about 1/4, the amount of 
calculation can be more reduced than a simple multistep search method. 

25 [0010] 

However, the multistep search method assumes such a property that 
the degree of similarity smoothly changes with respect to the positions of 
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the reference image regions as a precondition for accurately executing 
matching. Accordingly, when an image having many fine pictures and 
edges and an image including many noises are input, the precondition is not 
established, and thus an image region whose degree of similarity is not 
5 small may be detected in the matching at a first step. 
[0011] 

Fig. 3 shows an example of erroneous detection. A graph shows the 
degree of similarity that can be obtained from a template image and a 
reference image. Two minimal points exist in the example, and the right 

10 minimal point of them corresponds to the most similar reference image 

region as a correct answer. It is assumed that the left minimal point is a 
local minimum point less similar than the right one. When the multistep 
search method is applied to the example and the calculation of the degree of 
similarity at a first matching step is Hmited only to the positions shown by 

15 the large black circles in a broken line, the left minimal point, which is only 
a local minimum point, is erroneously detected. 
[0012] 

The erroneous detection is liable to occur when two or more minimal 
points of the degree of similarity exist, and the calculation of the degree of 

20 similarity that takes a minimum point is thinned out. In particular, when 
a degree of similarity containing the sum of the differences of respective 
pixels is used as in MAD, since the degree of similarity is increased even if it 
is slightly dislocated from a minimal point, an erroneous detection ratio is 
increased. 

25 [0013] 

An essential countermeasure for preventing the erroneous detection 
is to reduce the intervals of reference image regions whose degree of 



4 



r 




similarity is calculated at a first step or to select plural matching image 
regions at a first step and to execute matching in the peripheries of the 
plural image regions at a second step. However, since the number of times 
of calculation of the degree of similarity is increased, this countermeasure is 
5 not practical from the view point of the amount of calculation. 
[0014] 

As a method of improving the accuracy of matching at a fixst step 
without increasing the number of times of calculation of the degree of 
similarity, there is contemplated a method of interpolatingly estimating the 

10 degrees of similarity of the positions located among the positions, which are 
located at rough intervals and the degrees of similarity of which are 
calculated, from the calculated degrees of similarity. For example, the 
parametric template method of the non-patent document 2 improves 
detection accuracy at a first step by interpolating a correlation coefficient by 

15 expressing it by a quadratic continuous function. However, the method has 
a restriction in that the degree of similarity is hmited to the correlation 
coefficient. Further, a problem may arise in a load of a matrix calculation 
necessary to the interpolation. 
[0015] 

20 It is also contemplated to make use of degree of similarity 

interpolation methods called eqxxiangular fitting and parabola fitting, which 
are known as a degree of similarity interpolation method at a first step of 
the miiltistep search method. In the equiangular interpolation and the 
parabola interpolation, the degree of similarity calculated among three 

25 points is continuously interpolated by applying a broken line or a parabola 
that is symmetrical about a minimal point to the calculated degree of 
similarity (Fig. 14). These methods are used in a problem called subpixel 
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estimation for estimating a position with accuracy higher than the 
resolution of an input image, and the subpixel estimation is used to improve 
the accuracy of a position of a similar image after the position is found once 
with integer pixel accuracy (non-patent document 3). However, in a status 
5 in which erroneous detection occurs in a multistep search, the intervals at 
which a degree of similarity is calculated are insufficient as compared with 
the fineness of a template image, many noises are included, or the degree of 
similarity is not symmetrical at a minimal point in many cases. 
Accordingly, there is required a degree of similarity estimation method in 
10 which a local minimal solution does not fall even in the above cases. 
Non-patent Document l: AMurat Tekalp, "Digital Video Processing" 
Prentice Hall PTR,1995 

Non-patent document 2- K. Tanaka, M. Sano, S. Ohara, M. Okudara,"A 
parametic template method and its application to robust matching," in Proc. 
15 of IEEE Conference on Computer Vision and Pattern Recogntion, 2000. 
Non-patent document 3- Masao Shimizu and Masatoshi Okutomi, 
"Significance and Attributes of Sub-Pixel Estimation on Area-Based 
Matching", Systems and Computers in Japan, Vol. 34, No. 12, pp. 1-10, 
November 2003. 

20 

DISCLOSURE OF THE INVENTION 
[0016] 

As described above, the template matching executed by the all 
search method reqviires a large amount of the degree of similarity 
25 calculation. When the multistep search method is used, it is possible to 
more reduce the amount of calculation than the all search method. 
However, the multistep search method is defective in that an erroneous 
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image region is detected to an image including many fine pictures and edges 
and to an image containing many noises in a first step matching. It cannot 
be said that the degree of similarity interpolation method for preventing the 
erroneous detection has sufficient accuracy. 
5 [0017] 

Accordingly, an object of the present invention is to provide a 
template matching method having high detection accuracy with a small 
amount of calculation and an apparatus of it. In particular, it is an object 
of the present invention to provide a template matching method and an 
10 apparatus of it which reduce an error detection probability by accurately 
assuming a degree of similarity at a position where calculation is not 
executed even if an amount of calculation is reduced by roughly setting the 
position intervals in a degree of similarity calculation. 
[0018] 

15 According to the present invention, there is provided a template 

matching method of detecting the position of an image region similar to a 
template image region from on a reference image which is characterized by 
comprising calculating the degree of similarity of the image region to the 
template image region at rough position intervals, estimating the degree of 

20 similarity at a position where it is not calculated from the calculated degree 
of similarity making use of the gradients of a degree of similarity found 
independently in a plurality of directions, and determining the position 
having a small calculated or estimated value of the degree of similarity as 
the position of the similar image region. 

25 [0019] 

In the template matching method, the degree of similarity may be 
estimated making use of the degree of similarity estimated up to that time 
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in addition to the calculated degree of similarity in the estimation of the 

degree of similarity. 

[0020] 

-According to the present invention, there is provided a template 
5 matching method of narrowing down the position of a similar image region 
stepwise based on a multistep search method which is characterized by 
detecting the position of the similar image region by the template matching 
method before the final step of it. 
[0021] 

10 In the template matching method, the value region of the degree of 

similarity estimated value may be restricted such that the difference 
between the degree of similarity estimated value and the degree of 
similarity of a periphery used for the estimation or the gradient of the 
degree of similarity does not exceed a threshold value. 

15 [0022] 

In the template matching method, the threshold value may be 
determined based on an image region, which is obtained by moving the 
template image region in the same direction as or in the opposite direction 
to an estimated direction vector obtained by subtracting the position of the 
20 nearby image region from the position where the degree of similarity is 

estimated, and on the magnitude of the degree of similarity calculated from 

the template image. 

[0023] 

The template matching method may be characterized in that, when 
25 five reference image regions A, B, C, D, E are sequentially located on a 

straight hne, the degree of similarity at the position C is estimated from an 
interpolation value externally inserted assuming gradient continuity from 
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the degrees of similarity in the image regions A, B and from an interpolation 
value externally inserted assuming gradient continuity from the degrees of 
similarity in the image regions D, E in the estimation of the degree of 
similarity. 

5 

EFFECT OF THE INVENTION 
[0024] 

The present invention has an effect of providing a template 
matching method having a small amount of calculation and high detection 
10 accuracy. 

BRIEF DESCRIPTION OF THE DRAWINGS 
[00025] 

Fig. 1 is a schematic view explaining a degree of similarity 
15 estimation method of the present invention. 

Fig. 2 is a schematic view explaining a template matching method 
based on a conventional multistep search method. 

Fig. 3 is a schematic view showing a problem of the conventional 
multistep search method and an effect of the present invention. 
20 Fig. 4 is a schematic view showing the relation among a template 

image, a reference image, and a reference image region. 

Fig. 5 is a schematic view showing an example of a method of 
classifying a reference image region in an embodiment of the present 
invention. 

25 Fig. 6 is a block diagram showing an arrangement of the 

embodiment of the present invention; 

Fig. 7 is a sequence view showing a processing procedure of the 
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embodiment of the present invention. 

Fig. 8 is a schematic view showing the positions of reference image 
regions used for degree of similarity interpolation in the embodiment of the 
present invention. 

5 Fig. 9 is a schematic view explaining meaning of rounded threshold 

value calculation in the embodiment of the present invention. 

Fig. 10 is a schematic view showing a method of classifying a 
reference image region in the embodiment of the present invention. 
Fig. 11 is a block diagram showing an arrangement of the 
10 embodiment of the present invention. 

Fig. 12 is a table showing comparison of the amount of calculation of 
a conventional example with that of the embodiment of the present 
invention. 

Fig. 13 is a graph showing a result in which the embodiment of the 
15 present invention is applied to a motion estimation processing. 

Fig. 14 is a view showing an example of equiangular interpolation 
and parabola interpolation. 

DESCRIPTION OF REFERENCE NUMERALS 
20 [0026] 

601 storage of reference image 

602 storage of template image 

603 calculation of degree of similarity 

604 detection of degree of similarity minimizing candidate 
25 605 determination of rounded threshold value 

606 estimation of degree of similarity 

607 degree of similarity storage variable 
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BEST MODE FOR CARRYING OUT THE INVENTION 
[0027] 

A best mode for executing the present invention will be described 
5 below in detail with reference to the drawings. 
[0028] 

(Embodiment l) 

A first embodiment of the present invention will be explained below. 

Fig. 4 shows the relation among a function, a variable, and a coordinate 
10 used for explanation. S(i, j) shows a template image, and W(i, j) shows a 

reference image, (i, j) are a coordinate in an image and take integers in the 

embodiment. The position of a reference image region nominated as a 

matching candidate is shown by coordinates (u, v) in a reference image W (i, 

j). In the embodiment, (u, v) take integers. 
15 [0029] 

[Matching Evaluation Basis] 

The degree of similarity between the template image and the 

reference image region (u, v) is shown by e(u, v). It is assumed that 

smaller e(u, v) shows that the reference image region (u, v) is more similar 
20 to the template image. Template matching in the embodiment is a 

processing for detecting a reference image region (u, v) having small e(u, v). 

[0030] 

cmadCu, v) shown in an equation 1 is a specific example of the degree 
of similarity e(u, v). eMAoCu, v) is an mount showing the distance between 
25 two image regions by a difference absolute value average of pixel signals 

and called MAD (Mean Absolute Difference). In the equation 1, a group R 
shows the group of pixels that takes an average, and I R I shows the number 
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of elements of the group R. 
[Equation l] 
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[0031] 

5 [Outline of Matching Processing] 

In general, since the calculation of e(u, v) requires a calculation of a 
difference as many times as the number of pixels included in R, an amount 
of calculation is increased when the calcxilation is repeated many times. 
Thus, in the present invention, the amount of calculation of the matching 

10 processing is reduced by calculating the degree of similarity e(u, v) only to a 
part of reference image regions given as matching candidates. Instead of it, 
the position, where the calculation of the degree of similarity is omitted, the 
degree of the similarity is estimated by interpolating it from the calculated 
degree of similarity to prevent the drop of detection accuracy. 

15 [0032] 

For the purpose of explanation, a result that the degree of similarity 
of the reference image region (u, v) is estimated from the degree of 
similarity of a periphery is shown by estimate_e(u, v). The degree of 
similarity e(u, v) obtained as a result of calculation or as a result of 
20 estimation is shown by calc_e(u, v). 
[0033] 

A group of the reference image regions (u, v) from which the degree 
of similarity is actually calculated is shown by G(l). The reference image 
regions (u, v), which are estimated without calculating a degree of similarity, 
25 are classified to (K-l) pieces of groups G(2), G(3), ... , G(K) numbered in the 
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order in which an estimation processing is executed. At the time, it 
assumed that the degrees of similarity of the reference image regions (u, v) 
that belong to the same group are estimated by the same method. 
[0034] 

5 Although there are various methods of selecting the candidates G(l) 

whose degrees of similarity are to be calculated, it is preferable to select 
them by thinning out them so that the reference image regions in a search 
range can be entirely covered at a rough density Ukewise the first matching 
step of the multistep search method. It is preferable to classify candidates 
10 whose degrees of similarity are to be estimated to G(2), G(3), G(K) from 
the relative positional relation of them to the image regions that belong to 
(Gl). 
[0035] 

Parts (a), (b), (c) of Fig. 5 are views exemplifying a method of 
15 classifying the matching candidates to G(l), (G2), ... G(K). In the figure, it 
is assumed that a lattice pattern shows the coordinate s(u, v) of a reference 
image region and that one lattice box corresponds to one coordinate pixel. 
It is assumed that the reference image regions (u, v) labeled by 1 belong to 
(Gl) and that the reference image regions (u, v) labeled by 2, 3, 4 belong to 
20 G(2), G(3), G(4). The image regions that correspond to the boxes to which 
no numeral is labeled are removed from the matching candidates. 
[0036] 

In the example of the part (a) of Fig. 5, the regions labeled by "1" are 
disposed vertically and laterally at two pixel intervals. The degree of 
25 similarity is calculated to these "Is". Next, the degree of similarity of "2s" 
sandwiched laterally between "1" and "1" is estimated, the degree of 
similarity of "3s" sandwiched vertically between "1" and "1" is estimated, 
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and finally the degree of similarity of "4s" is estimated. In the examples of 
the parts (b) and (c) of Fig. 5, the regions labeled by "1" are disposed 
laterally at two pixel intervals. First, the degree of similarity of "Is" is 
calculated, and then the degree of similarity of the remaining candidates 
5 labeled by "2s" is estimated. 
[0037] 

Fig. 6 is a block diagram showing a template matching method in a 
first embodiment. A reference image W(i, j) and template data S(i, j) are 
input to image memories 601 and 602 firom the outside. 603 calculates the 

10 degrees of similarity e(u, v) of the matching candidates (u, v) belonging to 
G(l) and stores an obtained result in an array memory 607 as calc_e(u, v). 
605 calculates a threshold value clip_thresh(du, dv) for interpolation and 
roimding fi'om the template image S(i, j) and supphes a result to 606. 606 
estimates the degrees of similarity of (u, v) belonging to G(2), ... G(lO from 

15 the degree of similarity calc„e(u, v) stored in 607 and the rounding threshold 
value calculated in 605 and stores a result estimate_e(u, v) as calc_e(u, v). 
calc_e(u, v) output by 603 and 606 is also output to 604 which outputs a 
candidate (u*, v*) that minimizes calc_e(u, v) to the outside as a result of 
detection of the template matching. 

20 [0038] 

Fig. 7 shows a flow of a template matching processing in the first 
embodiment. At S701, the degrees of similarity e(u, v) of the reference 
image regions (u, v) belonging to the group G(l) of the reference images are 
calculated, and a result calc_e(u, v) is stored, (u, v) for minimizing calc_e(u, 
25 v) is also detected at the same time. Next, at S702, a rounded threshold 
value cUp_thresh(du, dv) used for the interpolation and estimation of the 
degree of similarity is calculated, and a result is stored to a variable. At 
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S703 to S705, the degrees of similarity of (u, v) belonging to G(l), G(2), ... 
G(K) are sequentially estimated, and a result of estimation estimate_e(u, v) 
is stored as calc_e(u, v). At the same time, (u, v) that minimizes calc_e(u, 
v) is also detected, and, at S706, finally obtained (u, v) is output as a result 
of matching. 
[0039] 

[Estimation of Degree of Similarity] 

A degree of similarity estimation method will be explained. It is 
necessary to accurately estimate a degree of similarity without causing an 
error even if minimum values locally exist in the degree of similarity 
calculated from a template image and a reference image as shown in Fig. 3 
as well as even if calculation of the degree of similarity in a most similar 
image is thinned out. At the time, a problem arises in that the degree of 
similarity abruptly changes in the vicinity of a minimal point. Since the 
degree of similarity has a different gradient on right and left sides when 
viewed from the minimal point, it is difficult to effectively execute 
estimation by an interpolation method that assumes smoothness of a curved 
line such as Unear interpolation, spUne interpolation, and the like. 
[0040] 

Accordingly, the present invention avoids a problem of gradient 
discontinuity at a minimal point by separately executing interpolation fi:om 
the degree of similarity of an image region located in a different direction 
when viewed from a point to be estimated. More specifically, the present 
invention increases detection accuracy by estimating the degree of similarity 
in two steps of a procedure for externally inserting an interpolation value or 
an interpolation cxirve making use the gradients of the degree of similarity 
independently found from plural directions and a procedure of determining 
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a finally estimated value by combining plural results of interpolation 
obtained. Here, the plural directions indicate one direction of left and right 
direction, one direction of up and down directions, and the like. It is not 
assumed that a gradient is symmetrical on a left side and a right side. An 
5 interpolation processing in the respective directions is executed by an 

ordinary method of the linear interpolation, the spline interpolation, and the 
like assuming that the gradient of the degree of similarity smoothly 
changes. 
[0041] 

10 For example, in Fig. 3, as to the degrees of similarity of the points 

without large black circle, interpolation curves are separately found from 
the degrees of similarity of the black points located on the left side and on 
the right side when viewed from the respective points, and the intersecting 
point of the thus obtained two interpolation curves is used as an degree of 

15 similarity estimated value. With this operation, the degree of similarity 
estimated value of the right side minimal point is smaller than that of the 
left side minimal point, thereby erroneous detection due to thinning out of 
the degree of similarity calculation can be prevented. 
[0042] 

20 The procedure of the degree of similarity estimation will be 

explained below in detail. In the present invention, the estimated value 
estimate_e(u, v) of the degree of similarity e(u, v) in the reference image 
region (u, v) is determined by the two steps of procedures, that is, a 
procedure of finding M pieces (M > 2) of a degree of similarity interpolation 

25 curve in the vicinity of (u, v) or an interpolation value I(u, vJ m) in (u, v) and 
a procedure of finding an estimated value estimate_e(u, v) from the obtained 
interpolation curve or interpolation value I(u, v; m). Here, m is an 
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identification number attached to the interpolation curve or the 
interpolation value. It is assumed that the interpolation value used to 
create the respective interpolation values I (u, vJ m) (m < M) is the 
interpolation value of the reference image region located in a different 
5 direction when viewed from (u, v) . 
[0043] 

Fig. 8 shows a specific example. In the figure, one lattice box 
corresponds to one pixel. Here, as to the degree of similarity estimated 
value estimate_e(P) of a reference image region P , first, I(P; l) is found from 

10 the degrees of similarity calc_e(All), calc_e(A12) of reference image regions 
All, A12, then I(P; 2) is found from calc_e(A2l), calc_e(A22), and finally the 
estimated value estimate_e(P) of e(P) is determined from I(P; l) and I(P; 2). 
Here, attention must be paid to that when All, A12 and A21, A22 are 
viewed from P, the directions where they are located are reversed between 

15 All, A12 and A21, A22. 
[0044] 

The degree of similarity estimation of a reference image region Q is 
the same as that of P. I(Q; 1) is calculated from the degrees of similarity of 
reference image regions Bll, B12, and then I(Q; 2) is calculated fi:om the 
20 degrees of similarity of B21, B22 located in a direction different from Bll, 
B12. Finally, estimate_e(Q) is determined from I(Q; l) and I(Q; 2). 
[0045] 

The degree of similarity estimation of a reference image region R is 
an example of calculation of four interpolation values. Eight points 
25 locating in the periphery of R is divided into 4 sets according to directions, 
and the degree of similarity of R is interpolated from respective two degrees 
of similarity. Specifically, I(R; l) is interpolated fi-om the degrees of 
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similarity of Cll, C12, 1(R; 2) is interpolated from the degrees of similarity 
of C21, C22, I(R; 3) is interpolated from the degrees of similarity of C31, C32, 
and I(R; 4) is interpolated from the degrees of similarity of C41, C42. A 
final estimated value estimate_e(R) is determined from the foxir 
5 interpolation values I(R; l), I(R; 2), I(R; 3), I(R; 4) found as described above. 
[0046] 

The respective interpolation curves or interpolation values I (u, vl m) 
to the degree of similarity of (u, v) are calculated assuming that the degree 
of similarity e(u, vm) has a continuously and gradually changing gradient in 
10 a region in which (u, v) and a reference image region used for interpolation 
exist. An ordinarily known method of interpolating a continuous function 
such as a linear interpolation, a sphne interpolation, and the like is used as 
an interpolation method. 
[0047] 

15 Fig. 1 is a graph showing an example of a method of interpolating 

and estimating the degree of similarity of the reference image region P from 
the degrees of similarity All, A12, A21, A22. In the graph, the lateral axis 
shows the coordinate u of the reference image region, and the vertical axis 
shows a degree of similarity. Black circles show a calculated degree of 

20 similarity calc„e(u, v). I(P; l) is a curved line linearly interpolated from 
All, A12, and I(P; 2) is a curved line found from A21, A22 by the linear 
interpolation. I(P; i; P) and I(P; 2; P) show values taken by an 
interpolation curve at P, and lis(P) shows the degree of similarity on the 
intersecting point of two curves. 

25 [0048] 

As a specific example, when A12, All, A21, A22 are disposed at 
equal intervals as well as P is located at the center between Al and A22, I(P; 
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i; P) and I(P; 21 P) are as shown in equations 2, 3, 4. In the equation 4, (e2 
- el) is the gradient of the degree of similarity found from A12 and All. An 
interpolation value from a left side in P is determined by multiplying (e2 ■ 
el) by a ratio 1/2 of the distance between P - All and the distance between 
5 All - A12 and subtracting a resultant value from el. When positional 

intervals are different, the coefficient of the equation 4 is changed according 
to the distance ratios among P, All, A 12. 
[Equation 2] 

7(P;1;P) = fi (calc_e(Jll),calc_e(^12)) 

10 [Equation 3] 

I{P;2;P} = fi (calc_e(^21),calc_e(^22)) 



[Equation 4] 



£2 fl 
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[0049] 

15 The degree of similarity estimated value of P is determined from the 

values taken by the interpolation curves I(P; l) and I(P; 2) in the vicinity of 
P. Although the determination can be made by various methods, one 
method is to use, for example, the maximum value Imax(P), the minimum 
value Imin(P), and the average value lava(P) of I(P; 1; P) and I(P; 21 P) as an 

20 estimated amount. Otherwise, it is also one method to express the 

estimated value by the linear sum Imix(P) of Imax(P) and Imin(P). They 
will be shown by an equation 5 to an equation 8. 
[Equation 5] 

r (P) = max{/(P;l)(P),/(P;2X-P)} 
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[Equation 6] 

(P) = min{/(P;l)(P),7(P;2)(P)} 

[Equation 7] 



_ CP) + (P) 
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5 [Equation 8] 
[0050] 

It is also one example to use the degree of similarity Iis(P) at the 
intersecting point of two curved lines interpolated to P in place of the 
10 interpolation value in P. As a specific example, when A12, All, A21, A22 
are located at the equal intervals, the estimated values shown by equations 
9, 10 are given by calculating the intersecting point of linear lines linearly 
interpolated from the left and right sides. 
[Equation 9] 

15 If there is a intersecting point between All and A12- 

4(P) = /2 (calc_e(-41 l);calc__e(^12),calc_e(^21),calc_e(^12)) 

If there isn't a intersecting point between All and A12^ 

calc_e(^l 1) + calc_e(^21) 



2 



[Equation 10] 

^ . , 3ac + bd — 2ad -2bc 



a+c—b—d 



[0051] 



As a method of determining an estimated value from 4 interpolation 
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curves I(R; l), I(R; 2), I(R; 3), I(R; 4) as in R of Fig. 8, there is a method of 
rearranging values I(R; 1) (p), I(R; 2)(p), I(R; 3)(p), I(R; 4)(p) in P of the four 
interpolation curves in the order of larger values and making them to 
Isort(R; l), Isort(R; 2), Isort(R; 3), Isort(R; 4), and expressing these four 
5 values by Imix (R) in which these four values are mixed by a linear sum at a 
ratio of (1 - 01 - 02 - 03) : 01 : 02 : 03 (equation 11). 

[Equation 11] 
10 [0052] 

Even in an example in which interpolation is difficult because the 
degree of similarity has plural minimal values as shown in Fig. 3 and a 
gradient greatly changes in the vicinities of minimal points, it is possible to 
estimate a degree of similarity near to an actual value by estimating the 
15 degree of similarity as described above. As a result, it is possible to reduce 
an error detection probability. 
[0053] 

[Rounding of Interpolation Estimated Value for Preventing Erroneous 
Detection] 

20 When an estimation processing is executed from a degree of 

similarity calculated using an image including noise, an error detection 
probabihty may be increased. For example, in the example of Fig. 1, when 
the difference between the degrees of similarity of A12 and All is increased 
by the effect of noise at the time P does not exist in the reference image 

25 region in which the degree of similarity is minimized, the degree of 

similarity of P is estimated smaller than an actual value. As a result, a 
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probability of erroneously detecting that P is in an matching region is 
increased. This phenomenon is particularly a problem in a smooth image 
whose degree of similarity less changes. Thus, to prevent an erroneous 
estimation due to mixture of noise, an extremely small interpolation value is 
5 avoided by restricting the gradient of an interpolation curve. 



[0054] 



Specifically, the gradient of an m-th interpolation curve I(P; m) in 



the reference image region P is restricted to a given value or less, and r(P; 
m) of an equation 12 is used for the interpolation of a degree of similarity. 
10 In the equation 12, I(P; m; Q) shows an interpolation value in the reference 
image region Q located in the vicinity of P. Ami shows a nearby reference 



image region used for interpolation, and | | Q-A | | shows the distance 
between Q and A. clip__thresh (P, Ami) shows the upper limit of the 
difference between e(Ami) and estimate_e(P), and the upper limit of the 
15 gradient magnetic field is obtained by dividing clip_thresh (P, Ami) by the 



distance I IQ^Amil I between P and Ami- 



[Equation 12] 



f/(P;w;e),e(^„,) 



r(P;m;Q) =max< 



cIip_thresh(P, ) 
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[0055] 



A similar effect can be also obtained even by employing a method of 



rounding a finally estimated value estimate_e(P) to stabilize matching in 
place of restricting a gradient every plural pieces of interpolation curves. 
In an equation 13, estimate_e'(P) specifically shows it. In the equation 13, 



M shows the number of the interpolation curves. 
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[Equation 13] 

estmiate_e(P) = niax{estimate_e(P), 
e(Aii ) - clip_thresh(P, ), 

) - clip Jiiresli(P, A^j^ )} 

[0056] 

Although the rounded threshold value clip_thresh (P, A) may be set 
5 to a fixed value, its optimum value is different depending on the 

characteristics of an input image. In the embodiment, the rounded 
threshold value is determined based on the following study. 
[0057] 

hp_thresh(P, A) shows the maximum value of the difference between 
10 e(A) and e(P). Since an object of estimating the degree of similarity of the 
image region P is to check whether or not P is similar to the template image, 
it is contemplated that matching can be stabihzed by setting the difference 
between e(A) and c(P) that can be taken when the image region P is similar 
to the template image. 
15 [0058] 

An amount for evaluating matching between the image region P, 
which has the same shape as the template image region, and the image 
region Q is defined by expanding the definition of the degree of similarity 
e(P) and set to e(P, Q). (Note^ since this is defined to show the idea of the 
20 present invention, e (P, Q) may not be actually defined.) Fig. 9 shows the 
relation between the regions A, P on the reference image and the template 
image S. 
[0059] 
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Since e(P, Q) can be regarded as the distance between the image 
regions, the relation of an equation 14 is estabhshed by a trigonometric 
inequaHty. The equation 14 can be modified to an equation 15 in view of 
that e(S, P), e(S, A) have the same degree of similarity. Here, when a 
5 situation that the image region P is matched to the template image is 

assumed, the distance e(P, A) between P and A can be approximated by the 
distance e(S, S(a-p)) between the template image S and an image S(a-p) 
obtained by dislocating the template image S in an interpolating direction 
(P-->A). An equation 16 shows that e(S, T) approximately gives the upper 
10 limit of the difference between e(A) and e(P). 
[Equation 14] 

[Equation 15] 

eiA)-e(P)^e(F>A) 

15 [Equation 16] 

e(A) - e(P) < e(P, A) s e(S, T) 

[0060] 

Accordingly, in the embodiment, the rounded threshold value 
clip_thresh (P, A) used to interpolate the degree of similarity in the image 
20 region P from A is determined based on the template image, which is moved 
in the same direction as or the opposite direction to the difference vector (du, 
dv) obtained by subtracting P from A, and on the degree of similarity 
calculated firom the template image. 
[0061] 

25 In the interpolation of P of Fig. 8, a threshold value is determined as 
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shown in, for example, an equation 17, kO, Xl (> 0) are adjustment 
parameters. When pixels can be only partly obtained in a region T in 
which the template image is subjected to parallel movement, the pixels are 
corrected by a method of finding e(S, T) from a part of the pixels that can be 
5 calculated and multiplying the value by a given value, and the like. Note 
that the amount of dislocation of the template image from S to T need not be 
equal to (du, dv) as long as the dislocating direction is the same direction as 
or the opposite direction to (du, dv). In this case, the parameter Xlis 
adjusted according to the ratio of the distance between S and T and the 
10 distance between P and A. clip_thresh2 shown in an equation 18 is a 

specific example when cmad shown in the equation 1 is used as a degree of 

similarity. 

[Equation 17] 

cUp_threshl(P,^) = max{0,--;i^ + ^eiS^T)} 

15 [Equation 18] 

clip_tliresh 2{du^ dv) 

= max jo,-^ +7^2 + J " ' 

[0062] 

The rounded threshold value clip_thresh as described above is the 
function of the interpolating direction vector (du, dv) and the template 
20 image S(i j), and the reference image W(i,j) is not included in the equation. 
Accordingly, it is sufficient to calculate the rounded threshold value only 
once when template matching starts, and it is not necessary to recalculate it 
each time the position of the reference image region is changed. 
[0063] 
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[Summary] 

As described above, the embodiment reduces the amount of 
calculation while minimizing the deterioration of detection accuracy by 
thinning out the calculation of the degree of similarity at rough position 
5 intervals and by estimating the degree of similarity of the position where it 
is not calculated by interpolating it from the degree of similarity to the 
reference image region in a periphery. 
[0064] 

At the time, in view of the property that the gradient of the degree of 
10 similarity is greatly changed in the vicinity of a minimal point, the error 

detection probability is reduced by estimating the degree of similarity by the 
procedure of finding the interpolation ciirve or the interpolation value 
making use of the gradients, which can be found independently in the plural 
directions when viewed from an interpolation point, and determining the 
15 degree of similarity from the result of plural interpolations. 
[0065] 

Further, the embodiment prevents the estimation error of the degree 
of similarity due to noise mixed to an input image by determining the 
rounded threshold value from the image obtained by dislocating the 

20 template image in the same direction as or the opposite direction to the 

interpolating direction and from the magnitude of the degree of similarity of 
the template image and by restricting the value range of the interpolation 
value or the estiniated value of the degree of similarity so that it is not 
smaller than the value obtained by subtracting the threshold value from a 

25 nearby degree of similarity. 
[0066] 

When the embodiment is used in the first matching step of the 
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template matching based on the multistep search method, the embodiment 
can obtain a great effect while realizing both the reduction of the amount of 
calculation and detection accuracy. When, for example, the embodiment is 
compared with the multistep search method of calculating the degree of 
5 similarity of the reference image region belonging to G(l) of the part (a) of 
Fig. 5 in a first matching step, the erroneous detection probabihty can be 
reduced in the embodiment because it estimates the degrees of similarity of 
G(2), G(3), G(4). As compared with the conventional multistep search 
method, although the amount of calculation is increased to estimate the 
10 degree of similarity, the increase of it is negligible as compared with a load 
when the degree of similarity is accurately calculated from an image. 
[0067] 

(Embodiment 2) 

A second embodiment of the present invention will be explained. A 
15 template matching method in the second embodiment is not different from 
the first embodiment except that a reduced image is used to calculate a 
degree of similarity to reduce an amount of calculation and that the 
definition of the degree of similarity and a method of calculating a rounded 
threshold value is different as a result. Accordingly, only the difference 
20 between it and the first embodiment will be explained. 
[0068] 

To simphfy explanation, only a case that an image is reduced at a 
reduction ratio of 1/2 in both the vertical and lateral directions is assumed. 
Images created by reducing a template image S(i, j) and a reference image 
25 W(i, j) are shown by S2(i, j), W2(i, j). It is assumed that the pixels 

corresponding to a coordinate (i, j) after reduction correspond to a coordinate 
(2i, 2j) before reduction. 
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[0069] 

The degree of similarity between the template image and a reference 
image region (u, v) is shown by e(u, v) likewise the first embodiment. It is 
assumed that coordinatesCu, v) are not coordinates on the reduced image 
5 and shows a coordinate value on the image before reduction. In the 

embodiment, the degree of similarity of the reference image region (u, v) is 
calculated from the reduced image S2(i, j), W2(i, j). eMAD2(2u, 2v) in an 
equation 19 is an example of e(u, v). Here, both u, v are even numbers, and 
a case that u or v is an odd number is not defined. It is assumed that a 
10 group R2 is a group of pixels for evaluating a difference in the reduced 
template image. 



[Equation 19] 



[0070] 

15 (u, v) for calculating a degree of similarity and (u, v) for 

interpolating a degree of similarity are classified according to the pattern 
shown in the part (a) of Fig. 5. One lattice box corresponds to one pixel in 
the image before reduction. As to (u, v) belonging to G(l), both u and v are 
even numbers, and thus eMAD2(u, v) of the equation 19 can be calculated. 

20 As to (u, v) to which G(2), G(3), G(4) belong, it is assumed that the value of 
the degree of similarity found according to an estimation procedure of the 
present invention is regarded as the degree of similarity e(u, v) in (u, v). 
[0071] 

As an example of a degree of similarity estimation method of the 
25 embodiment, the embodiment executes estimation by the same method as 
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the first embodiment except that the definition of the degree of similarity is 
different from the first embodiment. Further, as another example, a 
rounded threshold value clip_thresh(du, dv) may be calculated fi:om the 
reduced template image S2(i, j). An example of it is shown by cUp_thresh3 
5 of an equation 20. Although clip_thresh3(du, dv) is calculated fi'om the 
difference between the image obtained by dislocating the reduced template 
image by (du, dv) and the template image, since the amount of dislocation of 
(du, dv) in the reduced image is made twice in the image before reduction, a 
correction for multiplying the average of obtained differences by 1/2 is 
10 executed. 

[Equation 20] 

clip_tbresh3(c/w, dv) 

= maxjo +|-^ j;^\S,(_i + du,j+dv)-S,(iJ) 

[0072] 

The template matching method of the second embodiment is 
15 executed by the same procedure as the first embodiment except the points 
described above. 
[0073] 

Although an effect is the same as the first embodiment, an amount 
of calculation is smaller than the first embodiment because the reduced 
20 image is used. The embodiment is effective when it is used in a first 

matching step of the multistep template matching using a reduced image. 
[0074] 

(Embodiment 3) 

A third embodiment of the present invention will be explained by a 
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method of showing a difference between it and the first embodiment. 
[0075] 

A template matching method in the third embodiment estimates a 
matching image region with 0.5 pixel accuracy. Accordingly, the 
5 coordinates (u, v) of a reference image region take values of 0.5 pixel unit. 
It is assumed that a degree of similarity is, for example, eMAD(u, v) shown 
in the equation 1. The reference image W(i, j) must be referred to by the 
0.5 pixel unit, which defines a pixel internally inserted by an appropriate 
method according to a purpose of use of template matching. 
10 [0076] 

In the third embodiment, (u, v) for calculating a degree of similarity 
and (u, v) for interpolating a degree of similarity are classified according to 
the pattern shown in the part (a) of Fig. 5. However, different from the 
first embodiment, it is assumed that one lattice box corresponds to 0.5 pixel 

15 in place of one pixel and that (u, v) belonging to G(l) have integer coordinate 
values. That is, in the embodiment, the degree of similarity is calculated to 
a reference image region (u,v) in which both u, v are integers, and the 
degree of similarity is interpolatingly estimated to a reference image region 
(u,v) in which any of u, v has a non integer coordinate. 

20 [0077] 

A degree of similarity calculation method is the same as the first 
embodiment except the calculation of a rounded threshold value clip_thresh. 
As one method of calculating clip_thresh(du, dv), it is calculated based on 
the same definition as clip_thresh 2 of the equation 18. At the time, since 
25 (du, dv) are coordinate values less than integers, the template image S(i, j) 
must be referred to in the 0.5 pixel unit. However, S(i, j) is internally 
inserted by the same method as W(i, j). As another calculation method, the 
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threshold value is calculated by a calculation equation which is modified to 
refer to a pixel of an integer coordinate by doubling (du, dv) as 
clip_thresh4(du, dv) of an equation 21. 
[Equation 21] 

clip_thresli 4(du^ dv) 



< O -^l] 4 



= maxi 0 ~ + 2 + ^^^^ J + 2dv) - SQ, j}\ 

[0078] 

The template matching method of the third embodiment is executed 
by the same procedure as the first embodiment except the points described 
above. 
10 [0079] 

Since the embodiment can realize the template matching with 0.5 
pixel accuracy by the less number of times of degree of similarity calculation 
likewise the template matching with one pixel accuracy, it can execute 
matching with high accuracy as compared with the method of executing 
15 template matching rough with one pixel accuracy. The embodiment is 

effective when it is applied to a first matching step of the multistep search 
method for executing matching with 0.5 pixel accuracy in a second step. 



EXAMPLE 1 
20 [0080] 

As a first example of the present invention, there will be explained 
an example that makes use of the template matching method based on the 
present invention in a first matching processing step in a template matching 
processing based on a multistep search method using two-steps and a 
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reduced image. 
[0081] 

In the example, it is assumed that a template image S(i, j) is a 
rectangle having 16 x 16 pixels and that the range of coordinates i, j are 
5 integers from 0 or more to less than 16. It is assumed that a reference 
image W(i, j) is a rectangle having 48 x 48 pixels and that the range of 
coordinates i, j are integers from -16 or more to less than 32, It is assumed 
that the range of a reference image region (u, v) is such that both u, v are 
integers from -16 or more to 16 or less. The number of pieces of the 
10 reference image regions is 33 x 33 = 1089 pieces. 
[0082] 

At a first matching step, an image region whose degree of similarity 
is minimized is detected using an 8 x 8 template image S2(i, j) reduced to 1/2 
vertically and laterally and a 24 x 24 reference image reduced at the same 

15 ratio. Fig. 10 is a schematic view in which the reference image region (u,v) 
in the embodiment is shown by lattice -like boxes. In the figure, degrees of 
similarity are calculated to the reference image regions labeled by 1, and 
then degrees of similarity are interpolatingly estimated to the reference 
image regions labeled by 2, 3. 

20 [0083] 

It is assumed that a specific matching procedure is the same as the 
method explained in the second embodiment except that the reference image 
region is classified according to Fig. 10. It s assumed that the degree of 
similarity is defined by the equation 19. It is assumed that all the pixels 
25 included in the 8x8 template image belong to the group R2. The degree of 
similarity is interpolated from the degree of similarity of the reference 
image region having the same positional relation as the reference image 
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regions P, Q of Fig. 8, and an interpolation value is estimated according to 
the equations 2, 3, 4, 7. The interpolation value is rounded according to the 
equations 13 and 20. 
[0084] 

5 At a second matching step, the periphery of the image region 

detected at the first step is searched every second pixel while accurately 
calculating the degree of similarity using an image before reduction. A 
search range is set to 9 points in which the difference from the result of the 
first matching step is -1 or more to 1 or less in both the vertical and lateral 
10 directions. It is assumed that the degree of similarity is defined by the 
equation 1, 
[0085] 

Fig. 11 is a block diagram showing an arrangement of the example. 
A reference image is input to 1101, and a template image is input to 1102. 

15 The two input images are reduced to 1/2 vertically and laterally in 1103, the 
reduced reference image is stored in 1104, and the reduced template image 
is stored in 1105. 1106 executes a first step template matching processing 
from the reduced reference image of 1104 and the reduced template image of 
1105, and the coordinates (u, v) of a detected reference image region (u,v) 

20 are output to 1107. 1107 executes a second step template matching 

processing fi'om the reference image of 1101 and the template image of 1102 
and outputs the coordinates (u, v) of the reference image region (u, v) 
showing a result of final matching to the outside. 
[0086] 

25 An effect of the example will be shown by comparison. A 

conventional example A to which an all search method is applied and a 
conventional example B based on a multistep search method are prepared 
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as methods to be compared. The conventional example B executes 
matching by a procedure similar to this example except only a first step 
matching processing that is different from this example. In the 
conventional example B, a degree of similarity is not estimated in a first 
5 matching step and is calculated only to the reference image regions labeled 
by "1" in Fig. 10. 
[0087] 

A comparison of an amount of calculation will be shown. Fig. 12 
shows a result that an amount of calculation necessary to template 

10 matching is counted by the number of times of calculation of the difference 
of pixels. In the count, it is assumed that the difference calculation is 
executed 256 times when the degree of similarity is calculated by the image 
before reduction once, 64 times when the degree of similarity is calculated 
by a reduced image once, and 64 times when a threshold value rounded 

15 using the reduced image is calculated once. The amount of calculation does 
not include the amount of calcxilation of an estimation processing of other 
evaluation values, a minimum value detection processing, and image 
reduction is not included. It can be found from the result that the amount 
of calculation of the example is not almost different from the conventional 

20 example 2 based on the multistep search method and requires only an 
amount of calculation less than one thirteen the conventional example 1 
based on the all search method. 
[0088] 

A result of comparison obtained by an experiment of matching 
25 accuracy will be shown. To compare the accuracies of the template 

matching of the example, the conventional example A, and the conventional 
example B, the template matching processings of them were applied to the 
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estimation of the motion of a moving image. A progressive monochrome 
image having 720 x 480 pixels and 30 frames per second was used as an 
input image, and an image block for minimizing a degree of similarity in the 
block unit of 16 x 16 pixels was detected from a frame located just before a 
5 present frame with a time lag. To evaluate the matching accuracy, the 

value, which was obtained by averaging the minimum values of the degrees 
of similarity obtained by the template matching processing, was used as an 
index. A smaller index shows that the template matching is executed with 
a smaller amount of erroneous detection, and a larger index shows a higher 
10 erroneous detection probability. 
[0089] 

Fig- 13 are a set of graphs (a) and (b) showing an average degree of 
similarity obtained by estimating the motion of a to-be -evaluated moving 
image 13SD_Church. The graph (a) shows the average degrees of 

15 similarity of respective frames, and the lateral axis shows a frame number. 
Although the average degree of similarity of the example A is minimized in 
any of the frames, this is because the erroneous detection probabiUty is 
made to 0 due to the all search method. It can be found from the graph 
that the average degree of similarity of the conventional example B is 

20 greatly increased although this depends on a frame. In contrast, the 

example shows an average degree of similarity that is not greatly different 

from the conventional example A. 

[0090] 

Fig. 13(b) is a graph showing the ratio of the average degree of 
25 similarity of this example to that of the conventional example A, from which 
the difference between the average degree of similarity of this example and 
that of the conventional example A can be read outstandingly. The average 
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degree of similarity increased up to about 25% in the conventional example 
B is suppressed to about 5% in this example. As described above, this 
example can obtain matching accuracy which is more improved than the 
conventional example B based on the multistep search method and is near 
5 to the conventional example A based on the all search method. 
[009 IJ 

As shown above, in this example, it is possible to realize the 
template matching having accuracy near to the all search method in an 
amount of calculation that is not almost different from the conventional 
10 example based on the multistep search method. 
[0092] 

As described above, the degree of similarity between the template 
image and the matching reference image region is calculated only to the 
Umited reference image regions that are thinned out at the rough position 

15 intervals, and the degree of similarity at the position where it is not 

calculated is estimated from the calculated degree of similarity and the 
estimated degree of similarity making use of the gradients independently 
found in the plural directions. Accordingly, the matching processing can be 
executed by the small amount of calculation. 

20 [0093] 

Further, even if an image having a degree of similarity, which causes 
erroneous detection in the conventional multistep search method, is input, 
the erroneous detection probability can be prevented. 
[0094] 

25 Further, the value range of the degree of similarity estimated value 

is limited so that the difference between the degree of similarity estimated 
value and the degree of similarity of the periphery used for the estimation or 
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the gradient of the degree of similarity does not exceed a threshold value. 
As a result, even if a noise component is added to the reference image and 
the template image, an interpolation and estimation error caused by noise 
can be reduced, and thus matching accuracy can be increased. 
5 [0095] 

The threshold value is uniquely defined to interpolating direction 
vector that is obtained by subtracting the coordinates of the reference image 
region to be interpolated from the coordinates of the nearby reference image 
region. As a result, since it is sufficient to determine the threshold value 
10 once to each interpolating direction vector, it is not necessary to set the 
threshold value each time the position of the reference image region is 
changed, from which a necessary amount of calculation can be reduced. 
[0096] 

The threshold value is determined based on the value for evaluating 
15 the degree of similarity between the image region, which is obtained by 
moving the template image region in the same direction as or the opposite 
direction to to the interpolating direction vector, and the template image 
region. The threshold value determined by the method can prevent 
deterioration of the matching accuracy due to noise even if the noise 
20 component is added to the reference image and the template image because 
it can approximate the upper boundary of the magnitude of an evaluation 
value gradient. Further, since the threshold value is set to each 
interpolating direction vector, even if an image signal having a frequency 
characteristic different in a horizontal direction and a vertical direction is 
25 input, accurate matching can be executed. 
[0097] 

* 

The estimation of the degree of similarity by the interpolation is 
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executed only to the candidate regions located in the vicinity of the reference 
image region that locally minimizes the degree of similarity calculated or 
estimated up to a certain time. With this operation, since the amount of 
calculation necessary to estimate the degree of similarity can be reduced, it 
5 is possible to further reduce the amount of calculation of the template 
matching according to the present invention. 
[0098] 

The method explained above can be realized by a computer which 
reads a program from a program and executes it. 
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